Signal processing of audio and video data, including deriving identifying information

ABSTRACT

The present invention relates generally to signal processing of media content, e.g., audio or video content. One form of signal processing derives identifying information from media content. One claim recites a method including: upon encountering audio or video content, and using a programmed electronic processor, attempting to derive identifying information from data representing the encountered audio or video content; if a digital watermark is not obtained from the data representing the encountered audio or video content, modifying the encountered audio or video content; and then attempting to attempting to derive identifying information from data representing the modified audio or video content. Of course, other claims are provided too.

RELATED APPLICATION DATA

This application is a continuation of U.S. patent application Ser. No.11/545,081, filed Oct. 5, 2006 (now U.S. Pat. No. 7,602,940), which is acontinuation of U.S. patent application Ser. No. 10/379,393, filed Mar.3, 2003 (now U.S. Pat. No. 7,263,203). The 10/379,393 application is acontinuation of application Ser. No. 09/292,569, filed Apr. 15, 1999,which claims the benefit of U.S. provisional application No. 60/082,228,filed Apr. 16, 1998.

The Ser. No. 10/379,393 application is also a continuation ofapplication Ser. No. 09/998,763 (now U.S. Pat. No. 6,738,495), which isa division of application Ser. No. 09/292,569, filed Apr. 15, 1999,which claims the benefit of provisional application No. 60/082,228,filed Apr. 16, 1998.

The Ser. No. 10/379,393 application is also related to application Ser.No. 09/186,962, filed Nov. 5, 1998, which is a continuation ofapplication Ser. No. 08/649,419, filed May 16, 1996 (now U.S. Pat. No.5,862,260), which claims priority to application PCT/US96/06618, filedMay 7, 1996 (now published as WO9636163).

The Ser. No. 10/379,393 application is also related to application Ser.No. 09/442,440, filed Nov. 17, 1999, which is a continuation ofapplication Ser. No. 08/951,858, filed Oct. 16, 1997 (now U.S. Pat. No.6,026,193), which is a continuation of application Ser. No. 08/436,134,filed May 8, 1995 (now U.S. Pat. No. 5,748,763).

FIELD OF THE INVENTION

The present application relates to improvements in the field of digitalwatermarking.

BACKGROUND AND SUMMARY OF THE INVENTION

Digital watermarking (“watermarking”) is a quickly growing field ofendeavor, with several different approaches. The present assignee's workis reflected in the earlier-cited related applications, as well as inU.S. Pat. Nos. 5,841,978, 5,768,426, 5,748,783, 5,748,763, 5,745,604,5,710,834, 5,636,292, 5,721,788, and laid-open PCT applicationsWO97/43736 and WO99/10837 (corresponding to pending U.S. applicationSer. Nos. 08/746,613 (now U.S. Pat. No. 6,122,403) and 09/138,061).Other work is illustrated by U.S. Pat. Nos. 5,734,752, 5,646,997,5,659,726, 5,664,018, 5,671,277, 5,687,191, 5,687,236, 5,689,587,5,568,570, 5,572,247, 5,574,962, 5,579,124, 5,581,500, 5,613,004,5,629,770, 5,461,426, 5,743,631, 5,488,664, 5,530,759, 5,539,735,4,943,973, 5,337,361, 5,404,160, 5,404,377, 5,315,098, 5,319,735,5,337,362, 4,972,471, 5,161,210, 5,243,423, 5,091,966, 5,113,437,4,939,515, 5,374,976, 4,855,827, 4,876,617, 4,939,515, 4,963,998,4,969,041, and published foreign applications WO 98/02864, EP 822,550,WO 97/39410, WO 96/36163, GB 2,196,167, EP 777,197, EP 736,860, EP705,025, EP 766,468, EP 782,322, WO 95/20291, WO 96/26494, WO 96/36935,WO 96/42151, WO 97/22206, WO 97/26733. Some of the foregoing patentsrelate to visible watermarking techniques. Other visible watermarkingtechniques (e.g. data glyphs) are described in U.S. Pat. Nos. 5,706,364,5,689,620, 5,684,885, 5,680,223, 5,668,636, 5,640,647, 5,594,809.

Most of the work in watermarking, however, is not in the patentliterature but rather in published research. In addition to thepatentees of the foregoing patents, some of the other workers in thisfield (whose watermark-related writings can by found by an author searchin the INSPEC database) include I. Pitas, Eckhard Koch, Jian Zhao,Norishige Morimoto, Laurence Boney, Kineo Matsui, A. Z. Tirkel, FredMintzer, B. Macq, Ahmed H. Tewfik, Frederic Jordan, Naohisa Komatsu, andLawrence O'Gorman.

The artisan is assumed to be familiar with the foregoing prior art.

In the present disclosure it should be understood that references towatermarking encompass not only the assignee's watermarking technology,but can likewise be practiced with any other watermarking technology,such as those indicated above.

The physical manifestation of watermarked information most commonlytakes the form of altered signal values, such as slightly changed pixelvalues, picture luminance, picture colors, DCT coefficients,instantaneous audio amplitudes, etc. However, a watermark can also bemanifested in other ways, such as changes in the surface micro-topologyof a medium, localized chemical changes (e.g. in photographicemulsions), localized variations in optical density, localized changesin luminescence, etc. Watermarks can also be optically implemented inholograms and conventional paper watermarks.)

In accordance with the present invention, various improvements todigital watermarking are disclosed. For example, an improvedwatermarking method proceeds on an iterative basis in which thewatermark data is encoded in a source signal, the result is thendecoded, and the “strengths” of the individual encoded bits arediscerned. The watermarking parameters are then adjusted so as toredress any deficiencies determined in the first watermarking operation,and the source signal is re-watermarked—this time with the adjustedparameters. So doing assures reliable detection of all the componentbits.

The foregoing and other features and advantages of the present inventionwill be more readily apparent from the following Detailed Description,which proceeds with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1-9 are screen shots associated with an exemplary embodiment ofone aspect of the invention.

DETAILED DESCRIPTION

An improvement to existing watermark encoding techniques is to add aniterative assessment of the robustness of the mark, with a correspondingadjustment in a re-watermarking operation. Especially when encodingmultiple bit watermarks, the characteristics of the underlying contentmay result in some bits being more robustly (e.g. strongly) encoded thanothers. In an illustrative technique employing this improvement, awatermark is first embedded in an object. Next, a trial decodingoperation is performed. A confidence measure (e.g. signal-to-noiseratio) associated with each bit detected in the decoding operation isthen assessed. The bits that appear weakly encoded are identified, andcorresponding changes are made to the watermarking parameters to bringup the relative strengths of these bits. The object is then watermarkedanew, with the changed parameters. This process can be repeated, asneeded, until all of the bits comprising the encoded data areapproximately equally detectable from the encoded object, or meet somepredetermined signal-to-noise ratio threshold.

While the foregoing analysis evaluated confidence on a per-bit basis,related iterative procedures can evaluate confidence on a per-portionbasis. That is, the encoded object is considered in portions, and eachportion is analyzed for the robustness of the data encoded thereby. Inportions evidencing “weak” encoding, the encoding parameters can beadjusted to strengthen the encoding in one or more subsequentre-encoding operations.

The portions can take different forms, e.g., rectangular patches in astill or moving image; brief temporal excerpts in audio or video;certain DCT/Fourier/wavelet coefficients (or adjoining groups ofcoefficients) in coefficient-based representations of the object in atransformed domain, etc.

By this technique, even if the encoded object is spatially or temporallyexcerpted, or filtered (e.g. spectrally), there is increased assurancethat the watermark energy remaining after such processing will permitaccurate decoding.

In an illustrative embodiment, the process is highly automated andessentially transparent to a user. The user simply instructs acomputer-controlled system to watermark an object, and the systemresponds by performing the trial watermarking, decoding, makingsuccessive adjustments, and repeating as necessary until a final encodedobject meeting desired watermark-quality requirements is produced.

A more elaborate embodiment is a batch image processing system(implemented, e.g., in software run on a Pentium III-based personalcomputer system) that successively opens, watermarks, JPEG-compresses,and re-stores each image in a collection of images. Such systems arewell-suited for use by proprietors of large image collections (e.g. newsagencies, photo stock houses, etc.).

In such systems, any of the foregoing iterative-encoding techniques canbe employed. Or other iterative encoding techniques can be used. Oneother such other iterative encoding technique tests a trial encodingagainst one or more various forms of corruption, and makes adjustmentsbased on the results of the attempted decoding after such corruption(s).The process repeats as necessary.

In the exemplary system, the image is compressed after watermarking.Compression commonly weakens a watermark since some of the watermarksignal components are typically attenuated. The watermark can be mademore durable by increasing the watermark's energy, but so doing can havethe effect of visibly degrading image quality. Thus, a trade-off must bestruck between watermark durability and image fidelity.

A user-interface associated with the system can allow a user to selectone of three watermarking modes: low, medium, and high. In the low mode,image quality is more important than watermark durability. (Suchencoding is best-suited for applications in which the image is notexpected to be corrupted after distribution.) In the medium mode, imagequality and watermark durability are about equally important. In thehigh mode, watermark durability is more important than image quality.

Generally hidden from the user is a watermark intensity parameter—aparameter used by the watermarking algorithm to determine the intensity(e.g. amplitude) of the watermark signal added to the image. In theillustrative system, the parameter has a value between 1 and 16, with 1representing the lowest-intensity watermark, and 16 representing thehighest-intensity watermark. (The actual watermark energy is typicallynot a linear function of the parameter but can be, e.g., logarithmicallyrelated.)

The user interface can also allow the user to specify the degree of JPEGcompression desired (e.g. on an arbitrary low/medium/high, ornumerically defined scale), and indicate whether the compression can beadjusted—if necessary—to achieve the desired watermark durability/imagequality trade-off.

In operation, the system opens the first image file, and sets an initialintensity parameter (i.e. 1 to 16) corresponding to the user's modeselection (low, medium, or high). The resulting watermarked image isthen compressed in accordance with the user's instructions. A copy ofthe compressed file is next made and decompressed. The watermark is thendecoded from the decompressed image to check that the watermarked datacan be correctly read, and the strength of the watermark signal in theimage is assessed.

This strength metric (which may be, e.g., signal-to-noise ratioexpressed on a 1-10 scale) indicates the observed durability of thewatermark. If the correct watermark was read, the strength metric isthen compared against a threshold value corresponding to the selectedmode of operation. (I.e. if “low” mode is selected, a relatively lowthreshold is set on detected strength, whereas if “high” mode isselected, a relatively high threshold is set.)

If the strength of watermark detected in the decompressed image meets orexceeds the corresponding threshold value, the trial watermarking meetsthe user's objectives. Thus, the compressed/watermarked image is writtento a disk or other output storage device. Another file from thecollection is read, and the process repeats.

If the strength of the watermark detected in the decompressed image doesnot meet the corresponding threshold value (or if the correct watermarkwas not read), the system increases the intensity value and watermarksthe original image anew. The watermarked image is again compressed, anda copy made and decompressed. Again, the watermark is decoded from thedecompressed image to check for accuracy, and the strength of thewatermark signal in the image is assessed.

If the correct watermark was read, and the strength metric meets orexceeds the corresponding threshold, the watermarked/compressed image iswritten to the output storage device, and the process continues with anew image file.

If the strength of the watermark, or its readability, still don't meetspec, the intensity can be increased again and the process repeatedanew. However, at some point, the intensity parameter may reach a valueat which the image quality suffers unacceptably. Thus, the differentmodes of operation (low, medium, and optionally high) have correspondinginternal intensity threshold values beyond which the system will notincrease. If such threshold is reached, the system will not furtherincrease the embedded watermark intensity. Instead, it will next adjustthe compression (if the user has permitted the system—through theUI—this flexibility).

By lowering the degree of compression, encoded watermark energy betterpersists. An iterative watermark/compress/decompress/decode procedurethus begins in which the compression is successively backed-off untildecoding yields a watermark that meets the readability requirement andstrength threshold. (In an illustrated embodiment, the intensity remainsfixed at the highest level permitted for that mode while this iterativeprocedure proceeds.)

In some systems, a threshold is set beyond which the degree ofcompression cannot be lowered (again, for quality concerns). If thisthreshold is reached, and the decoded watermark still does not meet thereadability and strength tests, then the image is set-aside for latermanual consideration by the operator, and the process continues anewwith the next image. (The same type of exception handling occurs if theuser has not permitted the system to adjust the compression, and thestrength/readability tests cannot be met with watermark intensity withinthe permitted range.)

In these iterative systems, processing speed will depend on the size ofthe increments by which the parameters (e.g. encoding intensity,compression) are successively adjusted. In some embodiments, a finelevel of granularity is employed (e.g. changing the intensity by a unitvalue on successive iterations). In others, larger increments are used.In still others, binary search-type algorithms can be used to hone-in ona nearly-optimal value in a relatively short time.

The Figures show illustrative screen shots for use in an embodimentimplementing certain of the foregoing features. FIG. 1 shows an initialscreen, allowing the user to select between (1) starting the batchembedding; or (2) reviewing exceptions/errors from a previous batchembedding process and taking remedial actions (e.g., per FIG. 9 below).FIG. 2 shows the UI permitting the user to specify the mode of operationdesired (i.e., low/medium/high).

FIG. 3 shows a UI for selecting the desired degree of compression, whichcan be specified either as low, medium, or high, or by setting anumerical compression parameter. The UI of FIG. 3 also allows the userto request a warning if the compressed image file is larger than aspecified size, and permits the user to specify progressive scans.

FIG. 4 shows a UI permitting the user to specify the watermark payloadwith which the images are to be encoded. Among the fields are DigimarcID, Distributor ID, Do Not Copy flag, Restricted Use flag, Adult ContentFlag, Copyright Year, Image ID, and Transaction ID. The UI also permitsthe user to specify that certain IDs are to be incremented for eachsuccessive image in the batch.

FIG. 5 shows a UI permitting the user to specify the location of imagesto be processed. The UI optionally allows the user to preview images aspart of the process.

FIG. 6 shows a UI permitting the user to specify a destination folderfor the watermarked images, and output options (e.g. file format).

FIG. 7 shows a UI permitting the user to specify error and log filesettings (e.g. specifying maximum number of errors before stopping, andmaximum consecutive errors before stopping).

FIG. 8 shows a UI presenting errors and warnings to a user, e.g., aftera batch of image files has been processed. Included with the errorlistings are files that were not processed (e.g. because they were foundto already have a watermark). Included with the warning listings arefiles whose size, after compression, exceeded the earlier-specifiedmaximum file size (FIG. 3). The UI also permits the user to review allor selected files.

FIG. 9 shows a UI that presents an image for review, both after andbefore encoding/compression, and indicating the size of each image. Thescreen also includes UI controls permitting the user to adjust theintensity (on the 1 to 16 scale) and the JPEG compression level (eitheron a low/medium/high scale, or by numerical parameter). After the userhas made any desired changes, the “Next Image” button permits the nextimage to be displayed and processed.

(Although the just-described arrangement adjusted the intensity until athreshold was reached, and only then adjusted compression, in othersystems this need not be the case. For example, variations in intensityand compression might be alternated until a suitable result isachieved.)

While the just-discussed system checked robustness of the watermarkagainst JPEG-distortion (i.e. a lossy compression/decompressionprocess), the same procedure can be adapted to other types of expecteddistortion. For example, if an image is expected to be printed and thenscanned, such distortion might be modeled by down-sampling andblur-filtering. The system can check trial encodings against suchdistortion, and adjust the intensity (or other parameter) as needed toprovide a desired degree of durability against such corruption.

Likewise, while the just-discussed system checked trial encoding for twoperformance metrics (e.g., accurate decoding, and strength of watermarkmeeting a threshold), in other embodiments different performance metricscan be employed. Other sample metrics include, but are not limited to:speed of watermark decoding being within a threshold; change (i.e.increase or decrease) in bit-rate (e.g. for MPEG video or MP3/MP4 audio)being within a threshold; change in entropy (in the encoded vs.un-encoded object) being within a threshold, etc. (Changes in bit-rateand entropy resulting from watermarking are reviewed, e.g., in WO99/10837.)

In addition to testing for watermark robustness after imagecompression/decompression, the just-described embodiment also includedimage compression as part of the batch processing. Of course, this stepcould be omitted, and/or other processing steps (e.g. filtering,cropping, format translation, etc.) desired by the user could beincluded.

Watermarking can be applied to myriad forms of information. Theseinclude imagery (including video) and audio—whether represented indigital form (e.g. an image comprised of pixels, digital or MPEG video,MP3/MP4 audio, etc.), or in an analog representation (e.g. non-sampledmusic, printed imagery, banknotes, etc.) Watermarking can be applied todigital content (e.g. imagery, audio) either before or aftercompression. Watermarking can also be used in various “description” or“synthesis” language representations of content, such as StructuredAudio, Csound, NetSound, SNHC Audio and the like (c.f.http://sound.media.mit.edu/mpeg4/) by specifying synthesis commands thatgenerate watermark data as well as the intended audio signal.Watermarking can also be applied to ordinary media, whether or not itconveys information. Examples include paper, plastics, laminates,paper/film emulsions, etc. A watermark can embed a single bit ofinformation, or any number of bits.

The physical manifestation of watermarked information most commonlytakes the form of altered signal values, such as slightly changed pixelvalues, picture luminance, picture colors, DCT coefficients,instantaneous audio amplitudes, etc. However, a watermark can also bemanifested in other ways, such as changes in the surface micro-topologyof a medium, localized chemical changes (e.g. in photographicemulsions), localized variations in optical density, localized changesin luminescence, etc. Watermarks can also be optically implemented inholograms and conventional paper watermarks.

One improvement to existing technology is to employ established webcrawler services (e.g. AltaVista, Excite, or Inktomi) to search forwatermarked content (on the Web, in internet news groups, BBS systems,on-line systems, etc.) in addition to their usual datacollecting/indexing operations. Such crawlers can download files thatmay have embedded watermarks (e.g. *.JPG, *.WAV, etc.) for lateranalysis. These files can be processed, as described below, in realtime. More commonly, such files are queued and processed by a computerdistinct from the crawler computer. Instead of performing watermark-readoperations on each such file, a screening technique can be employed toidentify those most likely to be conveying watermark data. One suchtechnique is to perform a DCT operation on an image, and look forspectral coefficients associated with certain watermarking techniques(e.g. coefficients associated with an inclined embedded subliminalgrid). To decode spread-spectrum based watermarks, the analyzingcomputer requires access to the noise signal used to spread the datasignal. In one embodiment, interested parties submit their noise/keysignals to the crawler service so as to enable their marked content tobe located. The crawler service maintains such information inconfidence, and uses different noise signals in decoding an image (imageis used herein as a convenient shorthand for imagery, video, and audio)until watermarked data is found (if present). This allows the use of webcrawlers to locate content with privately-coded watermarks, instead ofjust publicly-coded watermarks, as is presently the case. The queuing ofcontent data for analysis provides certain opportunities forcomputational shortcuts. For example, like-sized images (e.g. 256×256pixels) can be tiled into a larger image, and examined as a unit for thepresence of watermark data. If the decoding technique (or the optionalpre-screening technique) employs a DCT transform or the like, the blocksize of the transform can be tailored to correspond to the tile size (orsome integral fraction thereof). Blocks indicated as likely havingwatermarks can then be subjected to a full read operation. If the queueddata is sorted by file name, file size, or checksum, duplicate files canbe identified. Once such duplicates are identified, the analysiscomputer need consider only one instance of the file. If watermark datais decoded from such a file, the content provider can be informed ofeach URL at which copies of the file were found.

Some commentators have observed that web crawler-based searches forwatermarked images can be defeated by breaking a watermarked image intosub-blocks (tiles). HTML instructions, or the like, cause the sub-blocksto be presented in tiled fashion, recreating the complete image.However, due to the small size of the component sub-blocks, watermarkreading is not reliably accomplished.

This attack is overcome by instructing the web-crawler to collect thedisplay instructions (e.g. HTML) by which image files are positioned fordisplay on a web page, in addition to the image files themselves. Beforefiles collected from a web page are scrutinized for watermarks, they canbe concatenated in the arrangement specified by the displayinstructions. By this arrangement, the tiles are reassembled, and thewatermark data can be reliably recovered.

Another such postulated attack against web crawler detection of imagewatermarks is to scramble the image (and thus the watermark) in a file,and employ a Java applet or the like to unscramble the image prior toviewing. Existing web crawlers inspect the file as they find it, so thewatermark is not detected. However, just as the Java descrambling appletcan be invoked when a user wishes access to a file, the same applet cansimilarly be employed in a web crawler to overcome such attemptedcircumvention of watermark detection.

Although “content” can be located and indexed by various web crawlers,the contents of the “content” are unknown. A *.JPG file, for example,may include pornography, a photo of a sunset, etc.

Watermarks can be used to indelibly associate meta-data within content(as opposed to stored in a data structure that forms another part of theobject, as is conventionally done with meta-data). The watermark caninclude text saying “sunset” or the like. More compact informationrepresentations can alternatively be employed (e.g. coded references).Still further, the watermark can include (or consist entirely of) aUnique ID (UID) that serves as an index (key) into a network-connectedremote database containing the meta data descriptors. The remote datamay contain meta-data described by an extensible markup language (XML)tag set. By such arrangements, web crawlers and the like can extract andindex the meta-data descriptor tags, allowing searches to be conductedbased on semantic descriptions of the file contents, rather than just byfile name.

Existing watermarks commonly embed information serving to communicatecopyright information. Some systems embed text identifying the copyrightholder. Others embed a UID which is used as an index into a databasewhere the name of the copyright owner, and associated information, isstored.

Looking ahead, watermarks should serve more than as silent copyrightnotices. One option is to use watermarks to embed “intelligence” incontent. One form of intelligence is knowing its “home.” “Home” can bethe URL of a site with which the content is associated. A photograph ofa car, for example, can be watermarked with data identifying the website of an auto-dealer that published the image. Wherever the imagegoes, it serves as a link back to the original disseminator. The sametechnique can be applied to corporate logos. Wherever they are copied onthe internet, a suitably-equipped browser or the like can decode thedata and link back to the corporation's home page. (Decoding may beeffected by positioning the cursor over the logo and pressing theright-mouse button, which opens a window of options—one of which isDecode Watermark.)

To reduce the data load of the watermark, the intelligence need not bewholly encoded in the content's watermark. Instead, the watermark canagain provide a UID—this time identifying a remote database record wherethe URL of the car dealer, etc., can be retrieved. In this manner,images and the like become marketing agents—linking consumers withvendors (with some visual salesmanship thrown in). In contrast to thecopyright paradigm, in which dissemination of imagery was an evil soughtto be tracked and stopped, dissemination of the imagery can now betreated as a selling opportunity. A watermarked image becomes a portalto a commercial transaction.

(Using an intermediate database between a watermarked content file andits ultimate home (i.e. indirect linking) serves an important advantage:it allows the disseminator to change the “home” simply by updating arecord in the database. Thus, for example, if one company is acquired byanother, the former company's smart images can be made to point to thenew company's home web page by updating a database record. In contrast,if the old company's home URL is hard-coded (i.e. watermarked) in theobject, it may point to a URL that eventually is abandoned. In thissense, the intermediate database serves as a switchboard that couplesthe file to its current home.

The foregoing techniques are not limited to digital content files. Thesame approach is equally applicable with printed imagery, etc. A printedcatalog, for example, can include a picture illustrating a jacket.Embedded in the picture is watermarked data. This data can be extractedby a simple hand-scanner/decoder device using straightforward scanningand decoding techniques (e.g. those known to artisans in those fields).In watermark-reading applications employing hand-scanners and the like,it is important that the watermark decoder be robust to rotation of theimage, since the catalog photo will likely be scanned off-axis. Oneoption is to encode subliminal graticules (e.g. visualizationsynchronization codes) in the catalog photo so that the set of imagedata can be post-processed to restore it to proper alignment prior todecoding.

The scanner/decoder device can be coupled to a modem-equipped computer,a telephone, or any other communications device. In the former instance,the device provides URL data to the computer's web browser, linking thebrowser to the catalog vendor's order page. (The device need not includeits own watermark decoder; this task can be performed by the computer.)The vendor's order page can detail the size and color options of thejacket, inventory availability, and solicit ordering instructions(credit card number, delivery options, etc.)—as is conventionally donewith on-line merchants. Such a device connected to a telephone can dialthe catalog vendor's toll-free automated order-taking telephone number(known, e.g., from data encoded in the watermark), and identify thejacket to the order center. Voice prompts can then solicit thecustomer's choice of size, color, and delivery options, which are inputby Touch Tone instructions, or by voiced words (using known voicerecognition software at the vendor facility).

In such applications, the watermark may be conceptualized as aninvisible bar code employed in a purchase transaction. Here, aselsewhere, the watermark can serve as a seamless interface bridging theprint and digital worlds

Another way of providing content with intelligence is to use thewatermark to provide Java or ActiveX code. The code can be embedded inthe content, or can be stored remotely and linked to the content. Whenthe watermarked object is activated, the code can be executed (eitherautomatically, or at the option of the user). This code can performvirtually any function. One is to “phone home”—initiating a browser andlinking to the object's home The object can then relay any manner ofdata to its home. This data can specify some attribute of the data, orits use. The code can also prevent accessing the underlying contentuntil permission is received. An example is a digital movie that, whendouble-clicked, automatically executes a watermark-embedded Java appletwhich links through a browser to the movie's distributor. The user isthen prompted to input a credit card number. After the number has beenverified and a charge made, the applet releases the content of the fileto the computer's viewer for viewing of the movie. Support for theseoperations is desirably provided via the computer's operating system, orplug-in software.

Similar functionality can be provided with an operating system totrigger special treatment of watermarked content, provided the operatingsystem is aware of the content type (e.g. in an object-oriented filesystem, or in a file system that is “media aware”). One exemplaryapplication is in the acquisition of content from externalaudio/image/video devices.

Such arrangements can also be used to collect user-provided demographicinformation when smart image content is accessed by the consumer of thecontent. The demographic information can be written to a remote databaseand can be used for market research, customization or personalization ofinformation about the content provided to the consumer, salesopportunities, advertising, etc.

In audio and video and the like, watermarks can serve to convey relatedinformation, such as links to WWW fan sites, actor biographies,advertising for marketing tie-ins (T-shirts, CDs, concert tickets). Insuch applications, it is desirable (but not necessary) to display on theuser interface (e.g. screen) a small logo to signal the presence ofadditional information. When the consumer selects the logo via someselection device (mouse, remote control button, etc.), the informationis revealed to the consumer, who can then interact with it.

Much has been written (and patented) on the topic of asset rightsmanagement. Sample patent documents include U.S. Pat. Nos. 5,715,403,5,638,443, 5,634,012, 5,629,980. Again, much of the technical work ismemorialized in journal articles, which can be identified by searchingfor relevant company names and trademarks such as IBM's Cryptolopesystem, Portland Software's ZipLock system, the Rights Exchange serviceby Softbank Net Solutions, and the DigiBox system from InterTrustTechnologies.

An exemplary asset management system makes content available (e.g. froma web server, or on a new computer's hard disk) in encrypted form.Associated with the encrypted content is data identifying the content(e.g. a preview) and data specifying various rights associated with thecontent. If a user wants to make fuller use of the content, the userprovides a charge authorization (e.g. a credit card) to the distributor,who then provides a decryption key, allowing access to the content.(Such systems are often realized using object-based technology. In suchsystems, the content is commonly said to be distributed in a “securecontainer.”)

Other asset management systems deliver content “in the clear”—notencrypted or in a secure container. Such arrangements are common, e.g.,in enterprise asset management systems (as opposed, for example, toe-commerce applications where container-based approaches are prevalent).

Desirably, the content should be marked (personalized/serialized) sothat any illicit use of the content can be tracked. This marking can beperformed with watermarking, which assures that the mark travels withthe content wherever—and in whatever form—it may go. The watermarkingcan be effected by the distributor—prior to dissemination of the(encrypted) object—such as by encoding a UID that is associated in adatabase with that particular container. When access is provided to thecontent (e.g. an access right is granted to a secure container, ornon-encrypted content is made available to a user), the database recordcan be updated to reflect the purchaser/user, the purchase date, therights granted, etc. An alternative is to include a watermark encoder inthe software tool used to access the content (e.g. a decryption engine,for secure container-based systems). Such an encoder can embed watermarkdata in the content before it is provided to the user (e.g. before it isreleased from the container, in secure container systems). The embeddeddata can include a UID, as described above. This UID can be assigned bythe distributor prior to disseminating the content/container.Alternatively, the UID can be a data string not known or created untilaccess (or the right to access) has been granted. In addition to theUID, the watermark can include other data not known to the distributor,e.g. information specific to the time(s) and manner(s) of accessing thecontent.

In still other non-container-based systems, access rights can again beimplemented realized with watermarks. Full resolution images, forexample, can be freely available on the web. If a user wishes toincorporate the imagery into a web page or a magazine, the user caninterrogate the imagery as to its terms and conditions of use. This mayentail linking to a web site specified by the embedded watermark(directly, or through an intermediate database), which specifies thedesired information. The user can then arrange the necessary payment,and use the image knowing that the necessary rights have been secured.

Tagging of image/video/audio assets at the time of their end-userdistribution (e.g. from a managed content repository) need not consistexclusively of a UID. Other data can also be watermarked into thecontent, e.g. a serial number, the identity of the recipient, the dateof distribution, etc. Additionally, the watermarked data can serve toestablish a persistent link to meta-data contained in an associatedasset management database.

As noted, digital watermarks can also be realized using conventional(e.g. paper) watermarking technologies. Known techniques forwatermarking media (e.g. paper, plastic, polymer) are disclosed in U.S.Pat. Nos. 5,536,468, 5,275,870, 4,760,239, 4,256,652, 4,370,200, and3,985,927 and can be adapted to display of a visual watermark instead ofa logo or the like. Note that some forms of traditional watermarks whichare designed to be viewed with transmissive light can also show up aslow level signals in reflective light, as is typically used in scanners.Transmissive illumination detection systems can also be employed todetect such watermarks, using optoelectronic traditional-watermarkdetection technologies known in the art.

As also noted, digital watermarks can be realized as part of opticalholograms. Known techniques for producing and securely mountingholograms are disclosed in U.S. Pat. Nos. 5,319,475, 5,694,229,5,492,370, 5,483,363, 5,658,411 and 5,310,222. To watermark a hologram,the watermark can be represented in the image or data model from whichthe holographic diffraction grating is produced. In one embodiment, thehologram is produced as before, and displays an object or symbol. Thewatermark markings appear in the background of the image so that theycan be detected from all viewing angles. In this context, it is notcritical that the watermark representation be essentially imperceptibleto the viewer. If desired, a fairly visible noise-like pattern can beused without impairing the use to which the hologram is put.

Digital watermarks can also be employed in conjunction with labels andtags. In addition to conventional label/tag printing processes, othertechniques—tailored to security—can also be employed. Known techniquesuseful in producing security labels/tags are disclosed in U.S. Pat. Nos.5,665,194, 5,732,979, 5,651,615, and 4,268,983. The imperceptibility ofwatermarked data, and the ease of machine decoding, are some of thebenefits associated with watermarked tags/labels. Additionally, the costis far less than many related technologies (e.g. holograms). Watermarksin this application can be used to authenticate the originality of aproduct label, either to the merchant or to the consumer of theassociated product, using a simple scanner device, thereby reducing therate of counterfeit product sales.

Conveniently ignoring for the time being the inevitable socialramifications of the following apparatus, consider a steganographicsetup whereby a given computing system which is based on a “masterclock” driving function can, with a small modification, impress a signalupon its electrical operations that could uniquely identify thecomputing system when that system communicates digitally with a secondcomputing system. The slight modification referred to would be such thatit would not interfere at all with the basic operation of the firstcomputing system, nor interfere with the communications between the twosystems.

The social ramifications referred to allude to the idea that there stillexists the notion of pure anonymity within an internet context and theassociated idea of the right to privacy. The “good” of this notion seemsto be that non-anonymity can be the seed for exploitation, while the“bad” of this notion seems to be an illusory escape from responsibilityfor one's own actions. Be this as it all may, there still seems to beground for the socially neutral concept of simply identifying a givenphysical object relative to its neighbors, hence the following approachto impressing a unique signal onto a computing system such that a secondsystem in communication with the first can identify the first.

Using spread spectrum principles (e.g. where a “noise like” carriersignal is modulated by an information signal to yield a data signal),the instantaneous phase of a computer's clock signal is modulated toconvey steganographically encoded data. In one embodiment, the phase ofthe clock frequency is instantaneously varied in accordance with thedata signal. (In some embodiments, the data signal needn't reflectmodulation with a noise-like carrier signal; the unaltered informationsignal can be sent.) In an illustrative system, the period of the clocksignal is 3 nanoseconds, and the instantaneous phase shifting is on theorder of +/−10 picoseconds. (These figures will, of course, depend onthe particular microprocessor used, and its tolerance for phase noisewhile still performing within specification.) If a binary “1” is to beencoded, the phase is advanced 10 picoseconds. If a binary “0” is to beencoded, the phase is retarded 10 picoseconds. Successive clock edgesare successively advanced or retarded in accordance with correspondingbits of the data signal until the end of data is reached. Then the datasignal desirably repeats, so as to provide redundant encoding.

Such modulation of the computer's clock system will be manifestedthroughout its operation, including in its communication with connecteddevices. Thus, for example, a modem or network connection will conveydata that is influenced by the phase modulation of the clock signal. Inparticular, the signal transitions in any such communication (e.g.RS-232, TCP/IP, wireless modem broadcasts, etc.) will not occur onuniformly-spaced clock edges, but instead will occur on clock edgeswhose exact temporal characteristics vary slightly in accordance withthe data signal with which the clock has been modulated.

A remote computer that receives information sent by such a computer canexamine the timing of the signal (e.g. its edge transitions), anddiscern thereby the encoded data signal.

If the channel to the remote computer offers near perfect temporalfidelity (i.e. it introduces essentially no variable delay into thedata's propagation), the data can be extracted by simply noting theinstantaneous offset of each edge transition from its expected nominalvalue, and a corresponding element of the data signal is therebyidentified. If the data signal was modulated by a noise-like signal,demodulation with the same noise-like signal will provide the originalinformation signal. (No such demodulation with a noise signal isrequired if the original information signal served as the basis for theclock phase shifting.)

More typically, the channel to the remote computer introduces temporaldistortion. An example is the internet, where different packets mayfollow different routings between the originating computer and thereceiving computer. In such environments, a large number of data must becollected and processed to extract the data signal. For example, it maybe necessary to average, or otherwise process, thousands of edgetransitions corresponding to a given bit in order to determine whetherthe edge was advanced or retarded, and thereby decode that bit with ahigh degree of statistical assurance. Thus, in environments with highlyvariable transmission latencies (e.g. the cited internet example), amessage of 100 bits might require collection and processing of on theorder of a million edge transitions to assure reliable message recovery.However, given the acceleration of data rates on the internet, this mayrequire only a few seconds of data.

In an illustrative embodiment, the master clock of the computer iscontrolled by a phase locked loop (PLL) circuit configured as a phasemodulator and driven by the data signal. Decoding is performed byanother PLL circuit, configured as a phase demodulator. (The two PLLsneedn't share a common time base, since decoding can proceed byexamining deviations of instantaneous edge transitions from the averageclock period.) The design of such PLL phase modulators and demodulatorsis within the routine capabilities of an artisan in this field.

Of course, the artisan will recognize that implementations not employingPLLs can readily be devised (e.g. employing other known phaseshifting/modulating techniques, delay circuits, etc.)

To provide a comprehensive disclosure without unduly lengthening thisspecification, applicant incorporates by reference the disclosures ofthe patents, laid-open international applications, and commonly-ownedU.S. applications cited herein.

Having described and illustrated a number of improvements to the fieldof digital watermarking in particular contexts, it will be recognizedthat the invention is not so limited. Instead, we claim as our inventionall such embodiments as come within the scope and spirit of thefollowing claims, and equivalents thereto.

1. A method comprising: upon encountering audio or video content,attempting to detect a digital watermark hidden therein with a processorof a computing device; if the digital watermark is not obtained from theencountered audio or video content, modifying the encountered audio orvideo content; and attempting to detect the digital watermark from themodified audio or video content.
 2. The method of claim 1, wherein apredetermined computer program is used to modify the encountered audioor video content.
 3. The method of claim 1, wherein modifying the audioor video content comprises descrambling the encountered audio or videocontent.
 4. A tangible computer readable medium having instructionsstored thereon, the instructions comprising: instructions to attempt todetect a digital watermark hidden within encountered audio or videocontent; instructions to modify the encountered audio or video contentif the digital watermark is not obtained from the encountered audio orvideo content; and instructions to attempt to detect the digitalwatermark from the modified audio or video content.
 5. A devicecomprising: a processor configured to: attempt to detect a digitalwatermark hidden within encountered audio or video content: modify theencountered audio or video content if the digital watermark is notobtained from the encountered audio or video content; and attempt todetect the digital watermark from the modified audio or video content.6. A method comprising: upon encountering audio or video content,attempting to derive identifying information from data representing theencountered audio or video content with a processor of a computingdevice; if a digital watermark is not obtained from the datarepresenting the encountered audio or video content, modifying theencountered audio or video content; and attempting to derive theidentifying information from data representing the modified audio orvideo content.
 7. The method of claim 6, wherein a predeterminedcomputer program is used to modify the encountered audio or videocontent.
 8. The method of claim 6, wherein modifying the encounteredaudio or video content comprises descrambling the encountered audio orvideo content.
 9. A tangible computer readable medium havinginstructions stored thereon, the instructions comprising: instructionsto attempt to derive identifying information from data representingencountered audio or video content; instructions to modify theencountered audio or video content if a digital watermark is notobtained from the data representing the encountered audio or videocontent; and instructions to attempt to derive the identifyinginformation from data representing the modified audio or video content.10. A device comprising: a processor configured to: attempt to deriveidentifying information from data representing encountered audio orvideo content; modify the encountered audio or video content if adigital watermark is not obtained from the data representing theencountered audio or video content; and attempt to derive theidentifying information from data representing the modified audio orvideo content.
 11. The method of claim 7, wherein modifying theencountered audio or video content comprises: collecting at least onecontent instruction; and concatenating a plurality of data blocks usingthe at least one content instruction to form modified audio or videocontent.
 12. The method of claim 11, wherein the at least one contentinstruction comprises display instructions, and wherein the displayinstructions identify positions of the plurality of data blocks on a webpage.
 13. The tangible computer readable medium of claim 4, wherein theinstructions to modify further comprise: instructions to collect atleast one content instruction; and instructions to concatenate aplurality of data blocks using the at least one content instruction toform modified audio or video content.
 14. The tangible computer readablemedium of claim 13, wherein the at least one content instructioncomprises display instructions, and wherein the display instructionsidentify positions of the plurality of data blocks on a web page. 15.The processor of claim 10, wherein the processor is further configuredto: collect at least one content instruction; and concatenate aplurality of data blocks using the at least one content instruction toform modified audio or video content.
 16. The processor of claim 15,wherein the at least one content instruction comprises displayinstructions, and wherein the display instructions identify positions ofthe plurality of data blocks on a web page.